Image processor for character recognition

ABSTRACT

In image processing of a document having image data of a document including an image with character images on a background image, the character images and the background image thereof are separated. First, areas in correspondence to the character images from the image data are extracted, and character code data are generated based on the extracted areas in the image data. On the other hand, the character image in the image data is replaced with the background image. The character images are reproduced with reference to the character code data at the positions of the character images. On the other hand, the original character images are deleted from the image data. The character images on the background image is complemented based on the background image data. Then the character images based on the character code data and the background image thereof are synthesized.

This application is based on application No. 2000-173727 filed in Japan,the contents of which are hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to optical character recognition whichconverts character images in image data to character code data.

2. Description of Prior Art

In character recognition by using an optical character recognitionapparatus, characters in a text image in a document are recognized. Asto a document including text image and the other image, it is known tofurther recognize the layout of the document and to arrange data of theother image at relevant positions in the layout as bit map image data.The layout, the character code data and the bit map image data areoutputted as one file.

However, when a document has characters on a background image, afterconverting the character image data to character code data, it isdifficult to synthesize the recognized characters (character imagesbased on the character code data) with the original image data. This isdue to difference in the font and the positions of the characters in theoriginal image data with the counterparts based on the conversion data.

In prior art character recognition, it is known, for example, as shownin FIG. 1, to convert the character image data in a document while notto output the background image. It is also known, as shown in FIG. 2, tosuperpose the character code data obtained by the conversion on theimage data of the document and to output the superposed image. However,in the former, the background image is not reproduced, and theinformation is lost partly. In the latter, the output image of thecharacter code data is liable to be shifted from that of the characterimage data, or the output image becomes obscure.

SUMMARY OF THE INVENTION

An object of the present invention is to provide image processing whichcan reproduce characters on the background image well.

In the present invention, character images and the background imagethereof are separated in image data of a document including an imagewith the character images on the background image. First, areas incorrespondence to the character images from the image data areextracted, and character code data are generated based on the extractedareas in the image data. On the other hand, the character image in theimage data is replaced with the background image. The character imagesare reproduced with reference to the character code data at thepositions of the character images. On the other hand, the originalcharacter images are deleted from the image data. The character imageson the background image is preferably complemented based on thebackground image data. Then the character images based on the charactercode data and the background image thereof are synthesized. Thus, thesynthesized image is reproduced well.

An advantage of the present invention is that a document image isreproduced well for character images on the background image.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects and features of the present invention willbecome clear from the following description taken in conjunction withthe preferred embodiments thereof with reference to the accompanyingdrawings, and in which:

FIG. 1 is a diagram of a prior art example of a document and an outputimage thereof;

FIG. 2 is a diagram of another prior art example of a document and anoutput image thereof;

FIG. 3 is a diagram of a system of an image recognition apparatus;

FIG. 4 is a diagram of an example of image data output;

FIG. 5 is a flowchart of image processing;

FIG. 6 is a diagram of a character with a circumscribing rectanglethereof;

FIG. 7 is a diagram of a maximum filter;

FIG. 8 is a diagram for explaining color detection;

FIG. 9 is a diagram of character deletion;

FIG. 10 is a diagram on the order of pixel positions for searchingcomplementary data;

FIG. 11 is a diagram for explaining trimming of image data; and

FIG. 12 is a diagram of a synthesized image.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to the drawings, wherein like reference charactersdesignate like or corresponding parts throughout the several views, FIG.3 shows a structure of a system according to a first embodiment of theinvention, wherein a computer 200 as a character recognition apparatushas a function of character recognition. The computer 200, a printer 300as an image output apparatus and a scanner 400 as an image inputapparatus are connected to a network 100. The computer 200 has a storagedevice 205 such as a random access memory or a hard disk for storingvarious programs and data, as well as a central processing unit (CPU)not shown in FIG. 3. The computer 200 processes the image data read bythe scanner 400. However, the processing may be performed similarly inthe scanner 400 or in the printer 300. Further, the invention may beapplied to a stand-alone digital copying machine.

The computer 200 has software programs such as a scanner driver 230 anda printer driver 240 which control the peripheral devices. For example,the scanner driver 230 instructs to set reading conditions and to startreading for the scanner 400, in a screen in the computer 200.

In a document recognition processor 210 in the computer 200, images onthe background image in a document are separated into the characterimages and the background bit map image. Characters are recognized inthe character images with use of optical character recognitiontechnique, while the character images are deleted from the image data.Then, they are synthesized again to reproduce the original document.

In this embodiment, the computer 200 has the function of the characterrecognition, but the character recognition function may also beincorporated in an image input apparatus or an image output apparatus.The image output apparatus is not limited to the printer 300, and it maybe, for example, a display device. The image input apparatus is notlimited to the scanner 400, and it may be a device such as a digitalcamera or a film scanner which inputs a digital image.

In the document recognition processor 210, a document recognitionprocessor 210 extracts character image data from image data which mayinclude a character image on a background image and converts theextracted character image data to character code data with opticalcharacter recognition. Further, the image data of the character image ona background image is complemented with reference to the ambientbackground image data. Then, image data based on the character code dataare synthesized with the complemented image data. Practically, acharacter recognizer 212 recognizes character image data in the imagedata obtained by reading a document and converts the character imagedata to character code data. A character deleter 214 deletes therecognized character code data from the original image data, and animage recognizer 216 recognizes image data such as a photograph or adesign other than the white space (or an area having no image). An imagesynthesizer 218 synthesizes the character code data with the image datarecognized by the image recognizer 216. Thus, the character image dataincluded in the original document does not overlap the character codedata. Further, by storing the character code data and the backgroundimage included in a document in the storage device 205, the characterimage data included in the document can be edited or changed by usingthe character code data. The character code data can also be used as keywords for search.

FIG. 4 shows document recognition by the document recognition processor210 schematically. A document having characters on the background imageis shown in the left side in FIG. 4. The image data of the charactercode data converted by the character recognizer 212 on the characterimage data in the document is shown in the lower portion in the rightside, while the upper portion in the right side shows the bit map imagedata after deleting the character image data in the document by thecharacter deleter 214 from the image recognized by the bit map imagerecognizer 216.

FIG. 5 is a flowchart of image processing in the document recognitionprocessor 210. First, image data of R (red), G (green) and B (blue)components obtained by reading a document by the scanner 400 areconverted to image data in a different color space of L, a and bcomponents independent of characteristics of a scanner device (S10).

Next, preprocessing for optical character recognition (OCR) is performedon the image data in the converted color space, in order to enhancerecognition efficiency in the character recognizer 212 (S12).Practically, the image data of L, a and b components are subjected tobinarization so that character image data become black and thebackground image becomes white. The image data of L, a and b componentsare stored in a different place (memory). Further, in the preprocessingof optical character recognition, for example, noises included in theimage data such as an isolated point may be deleted, inclination of theimage data due to misplacement of a document on document reading may becorrected, or deformed characters may be corrected. In this example, thepreprocessing is performed on the image data of L, a and b components,but it may be performed on the image data of R, G and B componentsbefore the conversion of color space.

Next, in the character recognizer 212, lines consisting of characterimage data included in the image data and each character image data inthe lines are taken out from the image data. Then, character code, font,font size and the like are recognized on each character image data(S14). The character code data is determined, for example, for eachcharacter in this embodiment. It may also be performed in the unit ofword. As to the font and font size, the most similar font and font sizeare selected among the font data group in the document recognitionprocessor 210. The character recognizer 212 outputs position coordinates(X1, Y1) and (X2, Y2) of a rectangle having the origin (0, 0) at the topleftmost point, as shown in FIG. 6, as position data of the recognizedcharacter image data.

When color is changed within a character, the character is regarded aresult of erroneous recognition on an image other than the character,and it is not converted to character code data. That is, character imagedata including color change is not converted to character code data. Inthe preprocessing for optical character recognition (S12), thebinarization is performed on the image data of L, a and b components,while the image data is stored in a different place. The color of acharacter is detected from the above-mentioned stored image data.Practically, by using the 3*3 maximum filter as shown in FIG. 7,filtering is performed to determine the maximum in the ambient 3*3pixels in the input monochromatic bi-level image for characterrecognition. Then, the character portion is contracted. This isperformed in order to delete influence of an edge of a character. Then,the image obtained by the contraction on the bi-level image and thecircumscribing rectangle obtained by character recognition are subjectedto an AND operation, and an average data on the pixels in the image isdetermined as the color of the character. That is, when C_(j) representscolor image data of the j-th character in the Lab space,

$\begin{matrix}{{C_{j} = ( {{\sum\limits_{i = 1}^{N}{{Li}/N}},{\sum\limits_{i = 1}^{N}{{ai}/N}},{\sum\limits_{i = 1}^{N}{{bi}/N}}} )},} & (1)\end{matrix}$wherein N denotes a number of pixels of the j-th character.

In the decision of color change, the circumscribing rectangle of acharacter after the subtraction is divided into four areas as shown inFIG. 8 with dashed lines, and the averages of L, a and b values in eacharea are compared. If the averages of L, a and b values in the fourareas are different by more than a threshold value, the color is decidedto be changed. If C_(j1) to C_(j4) represent the L, a and b values inthe four areas,C _(j1)=(L _(j1) , a _(j1) , b _(j1)),C _(j2)=(L _(j2) , a _(j2) , b _(j2)),C _(j3)=(L _(j3) , a _(j3) , b _(j3)),  (2)andC _(j4)=(L _(j4) , a _(j4) , b _(j4)).It is decided that the color is changed when the following condition issatisfied,|L _(jn) −L _(im) |≧k1,|a _(jn) −a _(jm) |≧k2,  (3)or|b _(jn) −b _(jm) |≧k3,wherein n=1, 2, 3 or 4, and m=1, 2, 3 or 4, and k1, k2 and k3 denoteconstants. The above-mentioned average image data C_(j) of the characteris determined actually by determining the averages of the four areasfirst and by further averaging the four averages. When the color ischanged in a character, the above-mentioned conversion to character codedata is not performed.

Next, in the character deleter 214, character image data recognized inthe character recognizer 212 is deleted from the original image data ofL, a and b components (S16). As mentioned above, the original image dataare stored in the different place in the binarization process in thepreprocessing at step S12. In the character deletion, the binarizedimage data are subjected to filtering with use of a 5*5 minimum filtershown in FIG. 9 in order to expand the character image data. Then, imagedata of L, a and b components in correspondence to the expandedcharacter image data are converted to white, as shown with a dashed linein FIG. 9.

Next, the image data of L, a and b components after the conversion towhite is subjected to filtering with a 7*7 filter shown in FIG. 10, sothat the image data are complemented according to the values of ambientpixels (S18).

An object pixel shown in FIG. 10 is a pixel in the image portion changedto white, and the numbers in the filter illustrate the order of theambient pixels for reference. According to the order in the filter, itis checked whether the pixel is a non-white pixel or not in the eightdirections of vertical, horizontal and oblique directions, and theobject pixel is corrected as an average of the first to third non-whitepixels. Thus, image data consisting only of the background image with nocharacter image are generated. Further, in order to decrease the amountof image data, only the necessary portion in the image data is extractedand stored. As shown in FIG. 11 schematically, the image data only ofthe background image is scanned sequentially from the top leftmostposition, and areas having densities larger than a threshold is trimmedas rectangular bit map image data as shown with dashed lines.

The bit map image data only of the background image generated asmentioned above and the character code data recognized in the characterrecognizer 212 are arranged and synthesized as shown in FIG. 12 (S20).The synthesis process depends on the format of output file. For example,in rich text format (RTF) or portable document format (PDF), the imagedata and character data are dealt as different objects. Then, as shownin FIG. 12, the image is arranged at the lower portion, while charactercode data overwrites the image for synthesis.

In the above-mentioned embodiment, characters on the background image ina document are separated into the character code data and the backgroundbit map image, from which characters are deleted, and they aresynthesized again to reproduce the document. Therefore, the system hasfollowing advantages.

(1) The encoded characters are not reproduced doubly with the characterimage, and the output image reproduces the background of the document.

(2) Further, because characters on the background can be encoded, theoutput character image can be read easily, the character size can bechanged, or the characters can be used for search.

(3) Still further, data in the background can be used again. Forexample, when the characters are modified, a document having the samebackground can be generated.

(4) Because character image data having a changing color is notconverted to character code data, erroneous recognition ofnon-characters as characters can be decreased.

Although the present invention has been fully described in connectionwith the preferred embodiments thereof with reference to theaccompanying drawings, it is to be noted that various changes andmodifications are apparent to those skilled in the art. Such changes andmodifications are to be understood as included within the scope of thepresent invention as defined by the appended claims unless they departtherefrom.

1. A method of image processing comprising the steps of: receiving imagedata of an original image, the original image having character imagesprovided on a background image; conducting character recognition on theimage data including preprocessing the image data for characterrecognition; extracting areas from the image data which correspond tothe character images; and generating character code data based on theextracted character images; changing, based on the characterrecognition, the image data by replacing the extracted areas with thebackground image with reference to the image data therein generatingchanged image data representing an image which is same as the originalimage except without the character images; and storing the changed imagedata and the character code data along with a relationship between them.2. The method according to claim 1, wherein, in the generating step, thecharacter code data are generated in reference to color information onthe character images.
 3. A method of image processing comprising thesteps of: conducting character recognition on image data includingpreprocessing the image data for character recognition; and convertingcharacter images in the preprocessed image data of an original image tocharacter code data; and complementing, based upon the characterrecognition, the character images based on image data around thecharacter images therein changing image data representing an image whichis same as the original image except without the character images. 4.The method according to claim 3, wherein, in the converting step, thecharacter code data are generated in reference to color information onthe character images.
 5. A method of image processing comprising thesteps of: conducting character recognition on image data includingpreprocessing the image data for character recognition; and convertingcharacter images in the preprocessed image data of an original image tocharacter code data; complementing, based upon the characterrecognition, the character images based on image data around thecharacter images therein generating changed image data representing animage which is same as the original image except without the characterimages; and storing the character code data and the changed image dataalong with a relationship between them.
 6. The method according to claim5, wherein, in the converting step, the character code data aregenerated in reference to color information on the character images. 7.An image processor comprising: a reader which reads an original image ofa document to provide image data thereof; character recognition deviceconducting character recognition on the image data including apreprocessing device preprocessing the image data for characterrecognition; a converter which determines character code data ofcharacter images in correspondence to character image in the image data;and an acquiring device which determines position data on a position inthe character images converted to character code data in the image data;a corrector which, based upon the character recognition, changes thecharacter images to the same as a color of an image around the characterimage with reference to the image data therein generating changed imagedata representing an image which is same as the original image exceptwithout the character images; and a storage device which stores thecharacter code data and the changed image data including thecomplemented character images along with a relationship between them. 8.The image processor according to claim 7, wherein said acquiring devicefurther determines font and font size based on the character images incorrespondence to the character image in the image data.
 9. The imageprocessor according to claim 7, further comprising a processor whichgenerates print data for printing the document image, based on thecharacter code data, the position data and the image data stored in saidstorage device.
 10. The image processor according to claim 7, whereinthe converter determines the character code data in reference to colorinformation on the character images.
 11. An image processor whichconverts character images in image data to character code datacomprising: character recognition device conducting characterrecognition on the image data including a preprocessing devicepreprocessing the image data for character recognition; an extractorwhich extracts character images in the preprocessed image data of anoriginal image, the original image having character images provided on abackground image; and a converter which converts the extracted characterimages to character code data; a deleter which, based on the characterrecognition, deletes the character images from the background image withreference to the image data therein generating changed image datarepresenting an image which is same as the original image except withoutthe character images; and a synthesizer which synthesizes the charactercode data with the changed image data.
 12. The image processor accordingto claim 11, wherein said deleter complements the image data at an areaof the character images on the background image according to image dataof an ambient background image of the area.
 13. The image processoraccording to claim 11, wherein said converter does not convert acharacter image to a character code data when an area of the characterimage has color change.
 14. The image processor according to claim 11,wherein said extractor extracts character images character by character.15. The image processor according to claim 11, wherein said extractorextracts the character images in the unit of word.
 16. The imageprocessor according to claim 11, wherein the converter converts theextracted character images to the character code data in reference tocolor information on the character images.
 17. A method of imageprocessing comprising the steps of: receiving image data of an originalimage, the original image having at least one character image on abackground image; conducting character recognition on the image dataincluding preprocessing the image data for character recognition; andgenerating character code data based on the at least one character imagewith reference to color information on the at least one character image;generating, based on the character recognition, changed image datarepresenting an image which is same as the original image except withoutthe at least one character image by replacing image data of the at leastone character image that has been converted into the character code datawith image data of the background image; and storing the changed imagedata and the character code data along with a relationship between them.